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Abstract 

In this paper we demonstrate that, ignoring computational constraints, it is possible to privately 
release synthetic databases that are useful for large classes of queries - much larger in size than the 
database itself. Specifically, we give a mechanism that privately releases synthetic data for a class of 
queries over a discrete domain with error that grows as a function of the size of the smallest net ap- 
proximately representing the answers to that class of queries. We show that this in particular implies a 
mechanism for counting queries that gives error guarantees that grow only with the VC-dimension of the 
class of queries, which itself grows only logarithmically with the size of the query class. 

We also show that it is not possible to privately release even simple classes of queries (such as in- 
tervals and their generalizations) over continuous domains. Despite this, we give a privacy-preserving 
polynomial time algorithm that releases information useful for all halfspace queries, given a slight re- 
laxation of the utility guarantee. This algorithm does not release synthetic data, but instead another data 
structure capable of representing an answer for each query. We also give an efficient algorithm for re- 
leasing synthetic data for the class of interval queries and axis-aligned rectangles of constant dimension. 

Finally, inspired by learning theory, we introduce a new notion of data privacy, which we call dis- 
tributional privacy, and show that it is strictly stronger than the prevailing privacy notion, differential 
privacy. 

1 Introduction 

As large-scale collection of personal information becomes easier, the problem of database privacy is increas- 
ingly important. In many cases, we might hope to learn useful information from sensitive data (for example, 
we might learn a correlation between smoking and lung cancer from a collection of medical records). How- 
ever, for legal, financial, or moral reasons, administrators of sensitive datasets might not want to release their 
data. If those with the expertise to learn from large datasets are not the same as those who administer the 
datasets, what is to be done? In order to study this problem theoretically, it is important to quantify what 
exactly we mean by "privacy." 

A series of recent papers IIDN04I IBDMN051 IDMNS061 formalizes the notion of differential privacy. A 
database privatization mechanism (which may be either interactive or non-interactive) satisfies differential 
privacy if the addition or removal of a single database element does not change the probability of any 
outcome of the privatization mechanism by more than some small amount. The definition is intended to 
capture the notion that "distributional information is not private" — we may reveal that smoking correlates to 
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lung cancer, but not that any individual has lung cancer. Individuals may submit their personal information 
to the database secure in the knowledge that (almost) nothing can be discovered from the database with their 
information that could not have been discovered without their information. 

In this paper, motivated by learning theory, we propose the study of privacy -preserving mechanisms that 
are useful for all queries in a particular class (such as all conjunction queries or all halfspace queries). In 
particular, we focus on counting queries of the form, "what fraction of the database entries satisfy predicate 
99?" and say that a sanitized output is useful for a class C if the answers to all queries in C have changed by 
at most some ita. 

Building on the techniques of Kasiviswanathan et al. IKLN+OSI . we show that for discretized domains, 
for any concept class that admits an a-net Ma, it is possible to privately release synthetic data that is useful 
for the class, with error that grows proportionally to the logarithm of the size of AAq. As a consequence, we 
show that it is possible to release data useful for a set of counting queries with error that grows proportionally 
to the VC-dimension of the class of queries. The algorithm is not in general computationally efficient. We 
are able to give a different algorithm that efficiently releases synthetic data for the class of interval queries 
(and more generally, axis-aligned rectangles in fixed dimension) that achieves guarantees in a similar range 
of parameters. 

Unfortunately, we show that for non-discretized domains, under the above definition of usefulness, it 
is impossible to publish a differentially private database that is useful for even quite simple classes such as 
interval queries. We next show how, under a natural relaxation of the usefulness criterion, one can release 
information that can be used to usefully answer (arbitrarily many) halfspace queries while satisfying pri- 
vacy. In particular, instead of requiring that useful mechanisms answer each query approximately correctly, 
we allow our algorithm to produce an answer that is approximately correct for some nearby query. This 
relaxation is motivated by the notion of large-margin separators in learning theory ||AB99[ Vap98[ ISS02I ; in 
particular, queries with no data points close to the separating hyperplane must be answered accurately, and 
the allowable error more generally is a function of the fraction of points close to the hyperplane. 

We also introduce a new concept, distributional privacy, which makes explicit the notion that when run 
on a database drawn from a distribution, privacy-preserving mechanisms should reveal only information 
about the underlying distribution, and nothing else. Given a distribution V over database points, a database 
privatization mechanism satisfies distributional privacy if with high probability, drawing an entirely new 
database from V does not change the probability of any outcome of the privatization mechanism by more 
than some small amount. We show that distributional privacy is a strictly stronger guarantee than differential 
privacy by showing that any mechanism that satisfies distributional privacy also satisfies differential privacy, 
but that there are some functions that can be answered accurately while satisfying differential privacy, and 
yet reveal information about the particular database (although not about any particular database element) 
that is not "distributional." 



1.1 Prior and Subsequent Work 
1.1.1 Prior Work 

Recent work on theoretical guarantees for data privacy was initiated by IIDN03i The notion of differential 
privacy, finally formalized by IIDMNS06II . separates issues of privacy from issues of outside information 
by defining privacy as indistinguishability of neighboring databases. This captures the notion that (nearly) 
anything that can be learned if your data is included in the database can also be leai^ned without your data. 
This notion of privacy ensures that users have very little incentive to withhold their information from the 
database. The connection between data privacy and incentive-compatibility was formahzed by McSherry 
and Talwar IIMT07L 
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Much of the initial work focused on lower bounds. Dinur and Nissim IIDN03II showed that any mecha- 
nism that answers substantially more than a linear number of subset-sum queries with error o{l/^/n) yields 
what they called blatant non-privacy - i.e. it allows an adversary to reconstruct all but a o(l) fraction of 
the original database. Dwork et al. IIDMT07II extend this result to the case in which the private mecha- 
nism can answer a constant fraction of queries with arbitrary error, and show that still if the error on the 
remaining queries is o(l/y^), the result is blatant non-privacy. Dwork and Yekhanin IIDY08I give further 
improvements. These results easily extend to the case of counting queries which we consider here. 

Dwork et al. IIDMNS06II . in the paper that defined differential privacy, show that releasing the answers 
to k low sensitivity queries with noise drawn independently from the Laplace distribution with scale k/e 
preserves e-differential privacy. Unfortunately, the noise scales linearly in the number of queries answered, 
and so this mechanism can only answer a sub-linear number of queries with non-trivial accuracy. Blum et 
al. IIBDMN05II consider a model of learning and show that concept classes that are learnable in the statisti- 
cal query (SQ) model ai^e also learnable from a polynomially sized dataset accessed through an interactive 
differential-privacy-preserving mechanism. We note that such mechanisms still access the database by ask- 
ing counting-queries perturbed with independent noise from the Laplace distribution, and so can still only 
make a sublinear number of queries. In this paper, we give a mechanism for privately answering counting 
queries with noise that grows only logarithmically with the number of queries asked (or more generally with 
the VC-dimension of the query class). This improvement allows an analyst to answer an exponentially large 
number of queries with non-trivial error, rather than only linearly many. 

Most similar to this paper is the work of Kasiviswanathan et al. [KL N+OSll . Kasiviswanathan et al. study 
what can be learned privately when what is desired is that the hypothesis output by the learning algorithm 
satisfies differential privacy. They show that in a PAC learning model in which the learner has access to the 
private database, ignoring computational constraints, anything that is PAC learnable is also privately PAC 
learnable. We build upon the technique in their paper to show that in fact, it is possible to privately release 
a dataset that is simultaneously useful for any function in a concept class of polynomial VC-dimension. 
Kasiviswanathan et al. also study several restrictions on learning algorithms, show separation between these 
learning models, and give efficient algorithms for learning particular concept classes. 

1.1.2 Subsequent Work 

Since the original publication of this paper in STOC 2008 IIBLR08I there has been a substantial amount of 
follow up work. A sequence of papers by Dwork et al. iDNR"'"09llDRV10ll give a non-interactive mechanism 
for releasing counting queries with accuracy that depends in a similar way to the mechanism presented in 
this paper on the total number of queries asked, but has a better dependence on the database size. This 
comes at the expense of relaxing the notion of e-differential privacy to an approximate version called (e, 6)- 
differential privacy. The mechanism of MDRVIOII also extends to arbitrary low-sensitivity queries rather 
than only counting queries. This extension makes crucial use of the relaxation to (e, (5)-privacy, as results 
such as those given in this paper cannot be extended to arbitrary low-sensitivity queries while satisfying 
e-differential privacy as shown recently by De BDelli 

Roth and Roughgarden MRRlOl showed that bounds similar to those achieved in this paper can also 
be achieved in the interactive setting, in which queries are allowed to arrive online and must be answered 
before the next query is known. In many applications, this gives a large improvement in the accuracy of 
answers, because it allows the analyst to pay for those queries which were actually asked in the course of 
a computation (which may be only polynomially many), as opposed to all queries which might potentially 
be asked, as is necessary for a non-interactive mechanism. Hardt and Rothblum HHRlOl gave an improved 
mechanism for the interactive setting based on the multiplicative weights framework which achieves bounds 
comparable to the improved bounds of HDRVIOI . also in the interactive setting. An offline version of this 
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mechanism (constructed by pairing the online mechanism with an agnostic learner for the class of queries 
of interest) was given by Gupta et al. MGHRUllll . and an experimental evaluation on real data was done 
by Hardt, Ligett, and McSherry MHLMllll . Gupta, Roth, and UUman unified the online mechanisms of 
IIRRlOi IHRlOl into a generic framework (and improved their error bounds) by giving a generic reduction 
from online learning algorithms in the mistake bound model to private query release algorithms in the 
interactive setting MGRUllll . BGRUllI also gives a new mechanism based on this reduction that achieves 
improved error guarantees for the setting in which the database size is comparable to the size of the data 
universe. 

Following this work, there has been significant attention paid to the problem of releasing the class 
of conjunctions (a special case of counting queries) with low error using algorithms with more efficient 
run-time than the one given in this paper. Gupta et al MGHRUllll give an algorithm which runs in time 
polynomial in the size of the database, and releases the class of conjunctions to 0(1) average error while 
preserving differential privacy. Hardt, Rothblum, and Servedio MHRSllI give an algorithm which runs 
in time proportional d'' (for databases over a data universe X = {0, l}*^) and releases conjunctions of 
most k variables with worst-case error guarantees. Their algorithm improves over the Laplace mechanism 
(which also requires run-time d'^) because it only requires that the database size be proportional to d^ (The 
Laplace mechanism would require a database of size d^). As a building block for this result, they also give a 
mechanism with run-time proportional to d^ which gives average-case error guarantees. Kasiviswanathan 
et al. UKRSUIOII extend the lower bounds IIDN03II from arbitrary subset-sum queries to hold also for an 
algorithm that only releases conjunctions. 

Xiao et al. HXWGIOI gave an algorithm for releasing range queries, which extends the class of constant- 
dimensional interval queries which we consider in this paper. 

There has also been progress in proving lower bounds. Dwork et al. llDNR+091 show that in general, the 
problem of releasing synthetic data giving non-trivial eiTor for arbitrary classes of counting queries requires 
run-time that is linear in the size of the data universe and the size of the query class (modulo cryptographic 
assumptions). This in particular precludes improving the run-time of the general mechanism presented 
in this paper to be only polynomial in the size of the database. Ullman and Vadhan HUVllll extend this 
result to show that releasing synthetic data is hard even for the simple class of conjunctions of at most 2 
variables. This striking result emphasizes that output representation is extremely important, because it is 
possible to release the answers to all of the (at most d"^) conjunctions of size 2 privately and efficiently 
using output representations other than synthetic data. Hardt and Talwar showed how to prove lower bounds 
for differentially query release using packing arguments, and gave an optimal lower bound for a certain 
range of parameters MHTIOI . De recently refined this style of argument and extended it to wider settings 
IIDelli Gupta et al. MGHRUllll showed that the class of queries that can be released by mechanisms that 
access the database using only statistical queries (which includes almost all mechanisms known to date, 
with the exception of the parity learning algorithm of MKLN"'"08| ) is equal to the class of queries that can be 
agnostically learned using statistical queries. This rules out a mechanism even for releasing conjunctions to 
subconstant eiTor which accesses the data using only a polynomial number of statistical queries. 



1.2 Motivation from Learning Theory 

From a machine learning perspective, one of the main reasons one would want to perform statistical analysis 
of a database in the first place is to gain information about the population from which that database was 
drawn. In particular, a fundamental result in learning theory is that if one views a database as a collection of 
random draws from some distribution V, and one is interested in a particular class C of boolean predicates 
over examples, then a database D of size 0(VCDIM(C)/a^) is sufficient so that with high probability, for 
every query q ^ C, the proportion of examples in D satisfying q is within ita of the true probability mass 
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under V |[AB99l |Vap98[ {l| Our main result can be viewed as asking how much larger does a database D 
have to be in order to do this in a privacy-preserving manner: that is, to allow one to (probabilistically) 
construct an output D that accurately approximates V with respect to all queries in C, and yet that reveals 
no extra information about database D]j In fact, our notion of distributional privacy (Section|7l) is motivated 
by this view. Note that since interactive privacy mechanisms can handle arbitrary queries of this form so 
long as only o(n) are requested, our objective is interesting only for classes C that contain Q{n), or even 
exponentially in n many queries. We will indeed achieve this (Theorem l3.10l) . since \C\ > 2^'^^^^'^^'> . 



1.3 Organization 

We present essential definitions in Section[2l In Section[3l we show that, ignoring computational constraints, 
one can release sanitized databases over discretized domains that are useful for any concept class with 
polynomial VC-dimension. We then, in Section IH give an efficient algorithm for privately releasing a 
database useful for the class of interval queries. We next turn to the study of halfspace queries over M*^ 
and show in Section [5] that, without relaxing the definition of usefulness, one cannot release a database that 
is privacy-preserving and useful for halfspace queries over a continuous domain. Relaxing our definition 
of usefulness, in Section [6l we give an algorithm that in polynomial time, creates a sanitized database that 
usefully and privately answers all halfspace queries. We present an alternative definition of privacy and 
discuss its relationship to differential privacy in Section |7J 

2 Definitions 

We consider databases which are n-tuples from some abstract domain X: i.e. D G X^. We will write 
n = I Z) I for the size of the database. We think of X as the set of all possible data-records. For example, if 
data elements ai^e represented as bit-strings of length d, then X = {0, 1}'^ would be the boolean hypercube in 
d dimensions. These tuples are not endowed with an ordering: they are simply multi-sets (they can contain 
multiple copies of the same element x G X). 

A database access mechanism is a randomized mapping A : X* — )• R, where R is some arbitrary range. 
We say that A outputs synthetic data if its output is itself a database: i.e. if i? = X*. 

Our privacy solution concept will be the by now standard notion of differential privacy. Crucial to 
this definition will be the notion of neighboring databases. We say that two databases D,D' G X"- are 
neighboring if they differ in only a single data element: i.e. they are neighbors if their symmetric difference 
\DAD'\ < 1. 

Definition 2.1 (Differential Privacy IIDMNS06I ). A database access mechanism A : X" — ^ R is e- 
differentially private if for all neighboring pairs of databases D,D' G X" and for all outcome events 
iS* C i?, the following holds: 

FiiA{D) eS]< exj>{e)FT[A{D') G S] 

In Section Ul we propose an alternate definition of privacy, distributional privacy, and show that it is 
strictly stronger than differential privacy. For simplicity, however, in the main body of the paper, we use 

'Usually, this kind of uniform convergence is stated as empirical error approximating true enor. In our setting, we have no 
notion of an "intrinsic label" of database elements. Rather, we imagine that different users may be interested in learning different 
things. For example, one user might want to learn a rule to predict feature Xd from features xi, . . . , Xd-i', another might want to 
use the first half of the features to predict a certain boolean function over the second half. 

"Formally, we only care about D approximating D with respect to C, and want this to be true no matter how D was constructed. 
However, if D was a random sample from a distribution D, then D will approximate V and therefore D will as well. 
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the standard definition, differential privacy. All of these proofs can be adapted to the distributional privacy 
notion. 



Definition 2.2. The global sensitivity of a query / is its maximum difference when evaluated on two neigh- 
boring databases: 

GSf= max \f(D)-f(D')\. 

In this paper, we consider the private release of information useful for classes of counting queries. 
Definition 2.3. A counting query Q^, defined in terms of a predicate 99 : X — )• {0, 1} is defined to be 

It evaluates to the fraction of elements in the database that satisfy the predicate if. 

Observation 2.4. For any predicate ip : X ^ {0, 1}, the corresponding counting query Q^p : — )• [0, 1] 
has global sensitivity GSq^ < ^/n 

Proof. Let D, D' G X'"- be neighboring databases. Then: 



\{xeD: vjx) = 1}\ ^ \{x e D' : y(x) = 1}| + 1 



Where the inequality follows by the definition of neighboring databases. Similarly, Qp{D) > Qp{D') — 
1/n. The observation then follows. □ 

We remark that everything in this paper easily extends to the case of more general linear queries, which 
can ai^e defined analogously to counting queries, but involve real valued predicates ip : X — )• [0, 1]. For 
simplicity we restrict ourselves to counting queries in this paper, but see URotlOl for the natural extension to 
linear queries. 

A key measure of complexity that we will use for counting queries is VC-dimension. VC-dimension is 
strictly speaking a measure of complexity of classes of predicates, but we will associate the VC-dimension 
of classes of predicates with their corresponding class of counting queries. 

Definition 2.5 (Shattering). A class of predicates P shatters a collection of points S C X if for every 
T C 5, there exists anip £ P such that {x £ S : (p{x) = 1} = T. That is, P shatters S if for every one of 
the 2l'5| subsets T of 5, there is some predicate in P that labels exactly those elements as positive, and does 
not label any of the elements in S" \ T as positive. 

We can now define our complexity measure for counting queries. 

Definition 2.6 (VC-Dimension). A collection of predicates P has VC-dimension d if there exists some set 
5 C X of cardinality 151=^ such that P shatters S, and P does not shatter any set of cardinality d + 1. 
We can denote this quantity by VC-DIM(P). We abuse notation and also write VC-DIM(C) where C is a 
class of counting queries, to denote the VC-dimension of the corresponding collection of predicates. 

Dwork et al. IIDMNS06II give a mechanism which can answer any single low-sensitivity query while 
preserving differential privacy: 
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Definition 2.7 (Laplace mechanism). The Laplace mechanism responds to a query Q by returning Q{D)+Z 
where Z is a random variable drawn from the Laplace distribution: Z ~ Lap(G5'Q/e). 

The Laplace distribution with scale b, which we denote by Lap (6), has probability density function 



Theorem 2.8 (Dwork et al. IIDMNS06II ). The Laplace mechanism preserves e-dijferential privacy. 

This mechanism answers queries interactively, but for a fixed privacy level, its accuracy guarantees 
degrade linearly in the number of queries that it answers. The following composition theorem is useful: it 
tells us that a mechanism which runs k e-differentially private subroutines is fce-differentially private. 

Theorem 2.9 (Dwork et al. IIDMNS06II ). If mechanisms Mi, . . . , are each e-differentially private, then 
the mechanism M defined by the (string) composition of the k mechanisms: M{D) = [Mi (D), . . . , Mk{D)) 
is ke-differentially private. 

We propose to construct database access mechanisms which produce one-shot (non-interactive) outputs 
that can be released to the public, and so can necessarily be used to answer an arbitrarily large number of 
queries. We seek to do this while simultaneously preserving privacy. However, as implied by the lower 
bounds of Dinur and Nissim IIDN03II . we cannot hope to be able to usefully answer arbitrary queries. We 
instead seek to release synthetic databases which are "useful" (defined below) for restricted classes of queries 
C. 

Definition 2.10 (Usefulness). A database access mechanism A is (a, 6)-useful with respect to queries in 
class C if for every database D G X", with probability at least 1—5, the output of the mechanism D = A{D) 
satisfies: 



In this paper, we will derive (a, (5)-useful mechanisms from small a-nets: 

Definition 2.11 (a-net). An a-net of databases with respect to a class of queries C is a set C X* such 
that for all D € X* , there exists an element of the a-net D' ^ N such that: 



We write Na{C) to denote an a-net of minimum cardinality among the set of all a-nets for C. 

3 General release mechanism 

In this section we present our general release mechanism. It is an instantiation of the exponential mechanism 
of McSheiTy and Talwar IIMT07i 

Given some arbitrary range TZ, the exponential mechanism is defined with respect so some quality 
function q : X* x 7^ — > M, which maps database/output pairs to quality scores. We should interpret 
this intuitively as a measure stating that fixing a database D, the user would prefer the mechanism to output 
some element of TZ with as high a quality score as possible. 

Definition 3.1 (The Exponential Mechanism IIMT07II ). The exponential mechanism ME{D,q,TZ) selects 
and outputs an element r £ TZ with probability proportional to exp(^|^^^). 




max\Q(D) - Q(D)\ < a 

Qec 



Ta&-i^\Q{D) - Q{D')\ < a 
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McSheny and Talwar showed that the exponential mechanism preserves differential privacy. It is im- 
portant to note that the exponential mechanism can define a complex distribution over a large arbitrary 
domain, and so it may not be possible to implement the exponential mechanism efficiently when the range 
of q is super-polynomially large in the natural parameters of the problem. This will be the case with our 
instantiation of it. 

Theorem 3.2 (" IIMT07II ). The exponential mechanism preserves e-dijferential privacy. 



Algorithm 1 The Net Mechanism 



NetMechanism(_D, C, e, a) 

Let7e^iVc,(C) 

Let g : X* X 7^ M be defined to be: 

q{D,D') = -max|Q(D) - Q{D')\ 

Sample And Output D' ^TZ with the exponential mechanism Me{D, q, TZ) 

We first observe that the Net mechanism preserves e-differential privacy. 
Property 3.3. The Net mechanism is e-dijferentially private. 

Proof. The Net mechanism is simply an instantiation of the exponential mechanism. Therefore, privacy 
follows from Theorem [321 □ 



We may now analyze the usefulness of the net mechanism. 

Property 3.4. For any class of queries C (not necessarily counting queries) the Net Mechanism is (2a, 5)- 
usefulfor any a such that: 

a> — log — - — 
e 

Where A = maxggc" GSq. 

Proof. First observe that the sensitivity of the quality score GSq < maxgec GSq = A. 

By the definition of an a-net, we know that there exists some D* ^ TZ such that q{D,D*) > —a. 
By the definition of the exponential mechanism, this D* is output with probability proportional to at least 
exjp{ ^g ). Similarly, there are at most |A^q,(C)| databases D' £ TZ such that q{D,D') < —la (simply 
because TZ = Na{C)). Hence, by a union bound, the probability that the exponential mechanism outputs 
some D' with q{D,D') < —2a is at most |A'^q,(C)| exp(^^|^). Therefore, if we denote by A the event 
that the net mechanism outputs some D* with q{D, D*) > —a, and denote by B the event that the net 
mechanism outputs some D' with q{D, D') < —2a, we have: 

Ft[A] ^ exp(^) 



Ft[B] - |Ar„(C)|exp(^) 
exp(ll) 

\Na{G)\ 

Note that if this ratio is at least 1/6, then we will have proven that the net mechanism is {2a, 6) useful with 
respect to C. Solving for a, we find that this is condition is satisfied so long as 

a> — log — J — 
e 
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□ 

We have therefore reduced the problem of giving upper bounds on the usefulness of differentially private 
database access mechanisms to the problem of upper bounding the sensitivity of the queries in question, and 
the size of the smallest a-net for the set of queries in question. Recall that for counting queries Q on 
databases of size n, we always have GSq < 1/n. Therefore we have the immediate corollary: 

Corollary 3.5. For any class of counting queries C the Net Mechanism is (2a, 6)-useful for any a such 
that: 

. 2 N^{C) 
a > — log — - — 
en 

To complete the proof of utility for the net mechanism for counting queries, it remains to prove upper 
bounds on the size of minimal a-nets for counting queries. We begin with a bound for finite classes of 
queries. 

Theorem 3.6. For any finite class of counting queries C: 

\Nc,{C)\ < \X\-^ 

In order to prove this theorem, we will show that for any collection of counting queries C and for any 
database D, there is a "small" database D' of size |-D'| = ^"^i*"^^ that approximately encodes the answers to 
every query in C, up to error a. Crucially, this bound will be independent of 

Lemma 3.7. For any D £ X* and for any finite collection of counting queries C, there exists a database 
D' of size 



such that: 



max\Q{D) - Q{D')\ < a 



Proof. Let m = We will construct a database D' by taking m uniformly random samples from the 

elements of D. Specifically, for i S {1, . . . , m} let Xj be a random variable taking value xj with probability 
\{x £ D : X = Xj}\/\D\, and let D' be the database containing elements Xi, . . . , X^. Now fix any Q^p G C 
and consider the quantity Q^{D'). We have: Q^{D') = — Yl^i ^i^i)- We note that each term of the sum 
ip{Xi) is a bounded random variable taking values < ip{Xi) < 1, and that the expectation of Q^p{D') is: 

nQ{D')] = -Y.n^{x.i)] = Q^{D) 

1=1 

Therefore, we can apply a standard Chernoff bound which gives: 

Pr [\Q^{D') - Q^{D)\ >a]< 2e-2-"' 
Taking a union bound over all of the counting queries Q^p G C, we get: 



Pr 



max \Q^{D') -Q^{D)\ > a 



< 2\C\e 



Plugging in m makes the right hand side smaller than 1 (so long as |C| > 2), proving that there exists a 
database of size m satisfying the stated bound, which completes the proof of the lemma. □ 
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Now we can complete the proof of Theorem 

Proof of Theorem By Lemma [3^ we have that for any D G X* there exists a database D' G X* with 
\D'\ = such that maxg^ec \Qv{D) - Q<^(^')l < Therefore, if we take N = {D' X* : \D'\ = 

} to be the set of every database of size , we have an a-net for C. Since 

log |C| 

|Ar| = \X\~^ 

and by definition \Na{C)\ < |A^|, we have proven the theorem. □ 
For infinite concept classes C, we can replace lemma 1377] with the following lemma: 



Lemma 3.8 ( IIAB99I Vap98[ ). For any D G X* and for any collection of counting queries C, there exists a 



database D' of size 

\D'\ = 0{VCDlM{C)\og{l/a)/a^) 



such that: 



max|Qp)-Q(Z?')| <« 



This lemma straightforwardly gives an analogue of Theorem 13. 6 1 
Theorem 3.9. For any class of counting queries C: 

\Na{C)\ < |X|<^(^<^^™('^)'°§(^/")/"') 

Note that we always have VCDIM(C) < log \ C\ for finite classes of counting queries, and so Theorem 
|3.9| is strictly stronger than Theorem 13.61 

Finally, we can instantiate Corollary 13. 5 1 to give our main utility theorem for the Net Mechanism. 

Theorem 3.10. For any class of counting queries C the Net Mechanism is {a, 5)-useful for any a such that: 
a>0 ( (VCDIM(C) log IXI log(l/a) + log 1/5) 
Solving for a, the Net Mechanism is (a, 5)-useful for: 



a = 



( VCDIM{C) log X + log 1 /6 



en 



1/3N 



Theorem 13. 101 shows that a database of size O ( '^^^F^^^'^'^ ) is sufficient in order to output a set of 
points that is a-useful for a concept class C, while simultaneously preserving e-differential privacy. If we 
were to view our database as having been drawn from some distribution V, this is only an extra 0{ ^°^^ ) 
factor larger than what would be required to achieve a-usefulness with respect to V, even without any 
privacy guarantee! 

The results in this section only apply for discretized database domains, and may not be computationally 
efficient. We explore these two issues further in the remaining sections of the paper. 
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3.1 The Necessity of a Dependence on VC-Dimension 

We just gave an e-differentially private mechanism that is (a, (5)-useful with respect to any set of count- 
ing queries C, when given a database of size n > 0{ ^°s^^cpiM{C) y ^^^^ section, we show that the 
dependence on the VC-dimension of the class C is tight. Namely: 

Theorem 3.11. For any class of counting queries C,for any < 5 < 1 bounded away from 1 by a constant, 
for any e < 1, if M is an e-differentially private mechanism that is (q, 5) useful for C given databases of 
size n < ^'^^™('^) ^ ij^gfi ct > Q ^ 4+i6e 

Proof. Fix a class of counting queries C corresponding to a class of predicates P of VC-dimension d. Let 
S C X denote a set of universe elements of size \S\ = d that are shattered by P, as guaranteed by the 
definition of VC-dimension. We will consider all subsets T C 5 of size |T| = d/2. Denote this set by 
Vs = {T C S : \T\ = d/2} For each such T G Vs, let tpr be the predicate such that: 



1, xeT; 
0, x^T. 



as guaranteed by the definition of shattering, and let Qt = Qipj. be the corresponding counting query. 
First we prove a simple lemma: 

Lemma 3.12. For every pair T, T' G Vs: 

I TAT' I 



Qt{T)-Qt{T') 



d 



Proof. 



Qt{T)-Qt{T') = 1,(y,Vt{x) 



(Pt{x) 



xGT' 




{iprix) - ipT{x)) + Y "Prix)- Y "^^^^^ 

x£T\T' xeT'\T 



d 

where the last equality follows from the fact that |T| = |r'|. □ 

This lemma will allow a sufficiently useful mechanism for the class of queries C to be used to reconstruct 
a database selected from Vs with high accuracy. 

Lemma 3.13. For any < 5 < 1 bounded away from 1 by a constant, let M be an (a, S)-useful mechanism 
for C. Given as input M (T) where T is any database T G Vs, there is a procedure which with constant 
probability 1 — 5 reconstructs a database T' with |T'AT| < 2da. 

Proof Write D' = M{T). With probabihty at least 1 - 5, we have maxggc \Q{T) - Q{D')\ < a. For the 
rest of the argument, assume this event occurs. For each T' G Vs, define: 

VT> =Qt'{T')-Qt'{D') 
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Let T' = argminj./g-pg vt'- We have: 

VT, <VT = Qt{T) - Qt{D') < a 

by the accuracy of the mechanism. By combining the accuracy of the mechanism with Lemma D.12[ We 
also have: 

I TAT' I 

VT' ^ Oi 

d 

Combining these two inequalities yields a database T' such that: |TAT'| < 2da as claimed. □ 

We can now complete the proof. Let T E Vs be a set selected uniformly at random, let x € T be an 
element of T selected uniformly at random, and let y € 5 \ T be an element of 5 \ T selected uniformly at 
random. Note that the marginal distributions on x and y are identical: both are uniformly random elements 
from S. Let T = {T \ {x}) U {y} be the set obtained by swapping elements x and y. Note that T is 
also distributed uniformly at random from Pg. Let T' be the set reconstructed from D' = M{T) as in 
lemma 13. 131 and let f' be the set reconstructed from D' = M(T). Assume that |rAT'| < 2da and that 
I TAT" I < 2da, which occurs except with probability at most 26. We now have: 

„ ITI - (l/2)|rAr'| f-da 
Pr[x G T'] = ' ' ^ ^ > = 1 - 2a 



2 



By symmetry: 



(l/2)|rAr'| da 



2 

Now recall that |TAT| < 2 and so by the fact that M is e-differentiaUy private, we also know: 

, , Prb G T'l 1 - 2a 1 

exp 2e > — ^ ^ > = 1 

~ Vt\x £ T'] ~ 2a 2a 

Because we also have exp(2e) < 1 + 8e, we can solve to find a > j^j^- □ 

4 Interval queries 

In this section we give an efficient algorithm for privately releasing a database useful for the class of interval 
queries over a discretized domain, given a database of size only polynomial in our privacy and usefulness 
parameters. We note that our algorithm is easily extended to the class of axis-aligned rectangles in d dimen- 
sional space for d a constant; we present the case of d = 1 for clarity. 

Consider a database D of n points in a discrete interval {1, . . . , 2"^} (in Corollarv 15.21 we show some 
discretization is necessary). Given ai < 02, both in {1,2,..., 2'^}, let Iai,a2 be the indicator function 
corresponding to the interval [ai, 02]. That is: 

T (^\- ! ai<x< 02; 
lauaA^) - I 0, Otherwise. 

Definition 4.1. An interval query Q[ai,a2] defined to be 

Q{a^M^^) - — nj\ — • 
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Algorithm 2 An algorithm for releasing synthetic data for interval queries. 
Releaselntervals(Z), a, e) 

Let ol ^ a/Q, Maxintervals ^ [4/3a'], e' ^ e/{d ■ Maxintervals). 
Let Bounds be an aiTay of length Maxintervals 
Let z 1, Bounds [0] <r- 1 
while Bounds [i - 1] < 2^^ do 

a <r- Bounds[i - 1], 6 ^ (2^^ - a + l)/2, increment ^ (2^^ - a + l)/4 
while increment > 1 do 

Let V ^ Qya,b\ (D) + Lap(l/(e'n)) 
if z) > a' then 

Let b b — increment 
else 

Let 6-^6 + increment 
end if 

Let increment ^ increment /2 
end while 

Let Bounds [i] 6, i i + 1 
end while 

Output D', a database that has a'm points in each interval [Bounds[j — 1], Bounds[j]] for each j € [i], 
for any m > 



Note that GSq^^^ = l/?i, and we may answer interval queries while preserving e-differential privacy 
by adding noise proportional to Lap(l/(en)). 

We now give the algorithm. We will assume for simplicity that all points x,x' ^ D aie distinct, but this 
condition can be easily discarded. The algorithm |4] is very simple. It repeatedly performs a binary search 
to partition the unit interval into regions that have approximately an a' fraction of the point mass in them. 
It then releases a database that has exactly an a' -fraction of the point mass in each of the intervals that it 
has discovered. There are at most ^ 1/a' such intervals, and each binary search terminates after at most d 
rounds (because the interval consists of at most 2'^ points). Therefore, the algorithm requires only d/a' 
accesses to the database, and each one is performed in a privacy preserving manner using noise from the 
Laplace mechanism. The privacy of the mechanism then follows immediately: 

Theorem 4.2. Releaselntervals is e-dijferentially private. 

Proof. The algorithm runs a binary search at most [4/3a'] times. Each time, the search halts after d queries 
to the database using the Laplace mechanism. Each query is l/e'-differentially private (the sensitivity of an 
interval query is 1/n since it is a counting query). Privacy then follows from the definition of e and the fact 
that the composition of k differentially private mechanisms is ke differentially private. □ 

Theorem 4.3. Releaselntervals is {a, 5)-useful for databases of size: 

8d , f8d\ 

n> log — 

ea \da J 

Proof. By a union bound and the definition of the Laplace distribution, if the database size n satisfies the 
hypothesis of the theorem, then except with probability at most 6, none of the {A/3)d/a' draws from the 
Laplace distribution have magnitude greater than a'^. That is, at every step, we have \v — Q[a,fe](-C)| < ct'^ 
except with probability /3. Conditioned on this event occurring, for each interval [Bounds[j — l],Bounds[j]] 
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for j G [i], /Bounds[j-i],Bounds[j](-^) ^ W ~ + a'^]- In the synthetic database D' released, each 

such interval contains exactly an a' fraction of the database elements. We can now analyze the error in- 
curred on any query when evaluated on the synthetic database instead of on the real database. Any in- 
terval [Bounds[j — l],Bounds[j]] C [a,b] will contribute eiTor at most a' to the total, and any interval 
[Bounds[j — l],Bounds[j]] ^ [a, b] that also intersects with [a, b] contributes enw at most (a' + a'^) to the 
total. Note that there are at most 2 intervals of this second type. Therefore, on any query Q[a,b] we have: 

\Q[a,b]iD') - Q[a,t]iD)\ < Yl \Q 

[Bounds [j — 1] , Bounds 

~ Q[Bounds[j-l],Bounds[j]] 

j : [Bounds [j - 1] , Bounds [j]] n [a,b]y^d 

< ±a" + 2{a' + a") 
6a 

< 6a 
= a 

□ 

We note that although the class of intervals (and more generally, low dimensional axis-aligned rectan- 
gles) is a simple class of functions, it nevertheless contains exponentially many queries, and so it is not 
feasible to simply ask all possible interval queries using an interactive mechanism. 



5 Lower bounds 

Could we possibly modify the results of Sections |4]and[3]to hold for non-discretized databases? Suppose we 
could usefully answer an arbitrary number of queries in some simple concept class C representing interval 
queries on the real line (for example, "How many points are contained within the following interval?") while 
still preserving privacy. Then, for any database containing single-dimensional real valued points, we would 
be able to answer median queries with values that fall between the 50 — 5, 50 + (5 percentile of database 
points by performing a binary search on D using A (where 6 = 6{a) is some small constant depending on 
the usefulness parameter a). However, answering such queries is impossible while guaranteeing differential 
privacy. Unfortunately, this would seem to rule out usefully answering queries in simple concept classes 
such as halfspaces and axis-aligned rectangles, that are generalizations of intervals. 

We say that a mechanism answers a median query M usefully if it outputs a real value r such that r falls 
within the 50 — 6,50 + 5 percentile of points in database D for some 5 < 50. 

Theorem 5.1. No mechanism A can answer median queries M with outputs that fall between the 50—6, 50+ 

6 percentile with positive probability on any real valued database D, while still preserving e-differential 
privacy, for 6 < 50 and any e. 

Proof. Consider real valued databases containing elements in the interval [0, 1]. Let Dq = {0, . . . ,0) be the 
database containing n points with value 0. Suppose A can answer median queries usefully. Then we must 
have 'Pi[A{Dq, M) = 0] > since every point in Dq is 0. Since [0, 1] is a continuous interval, there must 
be some value v G [0, 1] such that ¥t[A{Dq, M) = v] = 0. Let Z)„ = {v, . . . ,v)he. the database containing 
n points with value v. We must have Vi[A{Dn,M) = v\> 0. For 1 < i < n,\et Di = [0, . . . ,0,v, . . . ,v). 

n—i i 

Then we must have for some i, Pr[^(Z)j, M) = v] = but Pr[^(Z?j+i, M) = v] > 0. But since Di and 
Z^j+i differ only in a single element, this violates differential privacy. □ 



14 



Corollary 5.2. No mechanism can be {a, 6) -useful for the class of interval queries, nor for any class C 
that generalizes interval queries to higher dimensions (for example, half spaces, axis-aligned rectangles, or 
spheres), while preserving e-dijferential privacy, for any a, (5 < 1/2 and any e > 0. 

Proof. Consider any real valued database containing elements in the interval [0, 1]. \f A is (a, (5)-useful for 
interval queries and preserves differential privacy, then we can construct a mechanism A' that can answer 
median queries usefully while preserving differential privacy. By Theorem l5.1[ this is impossible. A' simply 
computes D = A{D), and performs binary search on D to find some interval [0, a] that contains n/2 it an 
points. Privacy is preserved since we only access D through A, which by assumption preserves e-differential 
privacy. With positive probability, all interval queries on D are correct to within ita, and so the binary search 
can proceed. Since a < 1/2, the result follows. □ 

We may get around the impossibility result of Corollary |5.2| bv relaxing our definitions. One approach is 
to discretize the database domain, as we do in Sections[3]and|4l Another approach, which we take in Section 
|6j is to relax our definition of usefulness. 



6 Answering Halfspace Queries 

In this section, we give a non-interactive mechanism for releasing the answers to "large-margin halfspace" 
queries, defined over databases consisting of n unit-length points in W^. The mechanism we give here will 
be different from the other mechanisms we have given in two respects. First, although it is a non-interactive 
mechanism, it will not output synthetic data, but instead another data structure representing the answers to 
its queries. Second, it will not offer a utility guarantee for all halfspace queries, but only those that have 
"large margin" with respect to the private database. Large margin, which we define below, is a property that 
a halfspace has with respect to a particular database. Note that by our impossibility result in the previous 
section, we know that without a relaxation of our utility goal, no private useful mechanism is possible. 

Definition 6.1 (Halfspace Queries). For a unit vector y G W^, the halfspace query fy : ^ {0, 1} is 
defined to be: 

^^^^ \ 0, Otherwise. 

Let Ch = {fy '■ y G IK'^JIylb = 1} denote the set of all halfspace queries. 

With respect to a database, a halfspace can have a certain margin 7: 

Definition 6.2 (Margin). A halfspace query fy has margin 7 with respect to a database D G (M'^)" if for all 
X e D: \ {x,y)\ > 7. 

Before we present the algorithm, we will introduce a useful fact about random projections, called the 
Johnson-Lindenstrauss lemma. It states, roughly, that the norm of a vector is accurately preserved with high 
probability when the vector is projected into a lower dimensional space with a random linear projection. 

Theorem 6.3 (The Johnson-Lindenstrauss Lemma IIDG99I IBBV06II ). For d > an integer and any < 
<f, r < 1/2, let A be a T X d random matrix with zbl/\/T' random entries, for T > 20<j~^ log(l/r). Then 
for any x G M*^.' 

Pr[|||A2;||2-||x||2| >?||x||i] <r 

For our purposes, the relevant fact will be that norm preserving projections also preserve pairwise inner 
products with high probability. The following corollary is well known. 
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Corollary 6.4 (The Johnson-Lindenstrauss Lemma for Inner Products). For d > an integer and any 
< <j, r < 1/2, let A be a T X d random matrix with ±1/ \/T random entries, for T > 20<j~^ log(l/ r). 
Then for any x € W^: 

Pv[\{{Ax), (Ay)) - {x,y)\ > '-{Ml + ||y||i)] < 2r 
A 2 

Proof. Consider the two vectors u = x + y and v = x — y. We apply Theorem [63] to u and v. By a union 
bound, except with probability 2r we have: + y)||2 — Ha^ + yllil + 2/111 ^i^^ lll^(^ ~ y)!!! ~ 

1 1^; — y| I2I < ?| — 2/| li- Therefore: 

{{Ax), {Ay)) = ^{{A{x + y),A{x + y)) - {A{x-y),A{x-y))) 

= \{\\A{x + y)\\l + \\A{x-y)\\l) 

< ^{{l + ,)\\x + y\\l-{l-,)\\x-y\\l) 

= {x,y) + l{\\x\\l + \\y\\l) 

An identical calculation shows that {{Ax), {Ay)) > {x,y) — | + Hylll)' which completes the proof. 

□ 

Instead of outputting synthetic data, our algorithm will output a datastructure based on a collection of 
random projections. 

Definition 6.5 (Projected Halfspace Data Structure). A T dimensional projected halfspace data structure of 

size m, Dh = {{Ai}, U, {vij}} consists of three parts: 

1. m independently selected random projection matrices Ai, . . . , Am G ffi^^*^ mapping vectors from 
to vectors in M^. 

2. A collection of T-dimensional unit vectors U C M"^ 

3. For each i G [m] and j ^ U, a. real number Vij G M. 

A projected halfspace data structure Dh can be used to evaluate a halfspace query fy as follows. We write 
fy{DH) to denote this evaluation: 

1. For i G [m], compute the projection yi G as: yi = Ai- y. 

2. For each i G [m] compute = argmin^. ^jj \\yi — Uj\\2 

3. Output fy{DH) = ^ 'ET=i 

What the projected halfspace data structure does is maintain m projections into a low dimensional space, 
as well as a collection of 'canonical' halfspaces U in T dimensions. The canonical halfspaces will be 
selected to form a net such that for every y G M?" with | |y| I2 = 1, there is some u £ Ui such that | |y — n| I2 < 
7/4. The size of U will be exponential in T, but we will choose T to be only a constant so that maintaining 
such a set is feasible. Each Vij will represent the approximate answer to the query /^^ on a projection of the 
private database by Ai. The Johnson-Lindenstrauss lemma will guarantee that not many points with margin 
7/2 ai^e shifted across the target halfspace by any particular projection, and the average of the approximate 
answers across all in projections will with high probability be accurate for every halfspace. 

First we bound the size of the needed net U for halfspaces. 
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Definition 6.6. A 7-net for unit vectors in is a set of points U C such that for all x G R-^ with 

min lla; — ylh < 7 



Claim 6.7. There is a 7-net for unit vectors in of size \ U\ < ■ 



Proof. Consider the set of T dimensional vectors discretized to the nearest multiple of 7/^^ in each co 



ordinate. There are (^^^ such vectors. For any unit x £ M^, let y = argmin^^gj^r \ \x — y\\2- We have: 

lk-y||2< ^EL(7/Vr)2 = 7. □ 

We can now present our algorithm. 

ReleaseHalfspaces(L>, d, 7, a, e) 
Let: 

? ^ 7 ^ ^ ? ^ ^ r20?-2 log(l/T)] m^^d (log(4Vd/7) + log(l//3) 

Let E { — l/-v/r, be a uniformly random matrix for each i € [m]. 

Let [/ be a 7/4-net for unit vectors in M^. 
for i = 1 to m do 

Let Di C be A = {^i^ : x e D}. 

for Xj G [/ do 

Letpij ^ Lap (^7^), ^ fx,{Di) + Pij 
end for 
end for 

Release Dh = ({AJ, C/, 



Theorem 6.8. ReleaseHalfspaces preserves e-differential privacy. 

Proof. Privacy follows from the fact that the composition of k e-differentially private mechanisms is ke- 
differentially private The algorithm makes m\U\ calls to the Laplace mechanism, and each call preserves 
e/ (m I [/ 1) -differential privacy (since each query has sensitivity 1/n). □ 

Theorem 6.9. For any database D with: 

n> log ^ j 

Then except with probability at most f3, Dh = ReleaseHalfSpaces{D, d, 7, a, e) is such that for each unit 
vector y G M'^ with margin 7 with respect to D: \fy{D) — fy{DH)\ < a. The running time of the algorithm 
and the bound on the size of D are both polynomial for 7, a G ^2(1). 

Proof. The analysis follows from the Johnson-Lindenstrauss lemma and a Chernoff bound. Let be a 
7/4-net for unit vectors in W^. Fix any y G R'^ such that fy has margin 7 with respect to D, and let y' = 
argmiiij^/g^d ||y — y'||2. Note that fyi has margin at least I7 with respect to D and that fy{D) = fyt{D). 
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Consider the quantity \fy/{D) — fyi{DH)\, where Dh is the projected halfspace data-structure output by 
ReleaseHalf spaces. By Corollary 16 .41 for each i G [m] and each x £ D: 

Fi[\{iAix),{Aiy')) - {x,y')\ > 7/4] < a/4 

By linearity of expectation, the expected number of points in D moved by more than 7/4 with respect to y' 
in any projection is at most an/4: 



E 



\{xeD: fy,{x) = fA,y'{Aix) A \{AiX,Aiy')\ > 



a 

> n I - - 



Let Ui^y' = argmin„g(y \\u — Aiy'\\2. Because \\ui^y' — Aiy'\\2 < 7/4, we have for any x G Di. 

\{Aiy ,x)\ = \{ui,x) + {Aiy' - ttiy,x)| < \{ui^y',x)\ + \\Aiy' - Uiy||2||x||2 < {ui,x) + 7/4 

Thus: 



E 



1 /a 

\{xe D: fy>{x) = fu^^^,{Aix) A \{AiX,Ui^y>)\ > -7}] > n [1 - - 



In other words, fy'{D) — a/4 < E[fu^ ^,{Di)] < fy'{D) + a/4. Moreover, for each i, fu. ^,{^i) 
independent random variable taking values in the bounded range [0, 1], and so we will be able to apply a 
Chernoff bound. For each y': 

i=l ^ ^ 

Taking a union bound over all (A^/d/j)'^ vectors y' G and plugging in our chosen value of m, and 
recalling our bound on E[fu^, (D)] we find that: 

Also note that the algorithm makes m\U\ draws from the distribution Lap ( ^^^M J during its run, assigning 



these draws to values pij. Except with probability at most /3/3, we have for all 

, , m|[/l f2m\U\\ 

Therefore, conditioning on \pij\ < 1 for all i,j and applying another Chernoff bound, we find that for any 
sequence of indices 

2- 



1 / 2 \ 

i=l 



Again taking a union bound and plugging in our value of m, we find that: 



Pr[ max I — Pi > a/41 < — 



Finally, conditioning on these three events (which together occur except with probability /3), we have for 



any y': 



1=1 \i=l 1=1 / 



+ a 

which completes the proof. □ 
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7 Distributional Privacy 



In this section we give an alternative privacy definition, motivated by learning theory, and show that it is a 
strengthening of differential privacy. 

Definition 7.1 (S'-neighbors). For any subset of the universe S* C X of size \S\ > n, we say that two 
databases are S'-neighbors if they were both drawn at random without replacement from S. 

Definition 7.2 (Distributional Privacy). We say that a mechanism A : X" — )• R satisfies (e, 13) -distributional 
privacy if for any 5 C X, and for any pair of databases Di,D2 C X of size n which ai^e S'-neighbors, with 
probability 1 — /3 over the draw of the databases we have for all events E (1 R: 

Pr[A(L»i) eE]< e^Vv[A[D2) G E]. 

For example, suppose that a collection of hospitals in a region each treats a random sample of patients 
with disease X. A hospital can release information with a guarantee of distributional privacy, which is 
informative about patients in the region, without revealing which hospital the data came from. Actually, our 
main motivation is that this definition is particularly natural from the perspective of learning theory: given 
a sample of points drawn from some distribution V, one would like to reveal no more information about the 
sample than is inherent in V itself. 

We will typically think of p as being exponentially small, whereas e must be r2(l/n) for ^ to be useful. 

7.1 Relationship Between Definitions 

It is not a priori clear whether either differential privacy or distributional privacy is a stronger notion than 
the other, or if the two are equivalent, or distinct. On the one hand, differential privacy only provides 
a guarantee when Di and D2 differ in a single elementJl whereas distributional privacy can provide a 
guarantee for two databases Di and D2 that differ in all of their elements. On the other hand, distributional 
privacy makes the strong assumption that the elements in Di and D2 are drawn from some distribution 
V, and allows for privacy violations with some exponentially small probability /3 (necessarily: with some 
small probability, two databases drawn from the same distribution might nevertheless possess very different 
statistical properties). However, as we show, distributional privacy is a strictly stronger guarantee than 
differential privacy. 

Theorem 7.3. If A satisfies (e, P) -distributional privacy for any fi = o(l/n^), then A satisfies e-differential 
privacy. 

Proof. Consider any database Di drawn from domain X, and any neighboring database D2 that differs from 
Di in only a single element x ^ X. Let S = Di U {x} be a set of size |S| = n + 1. Consider two S- 
neighbors D[ and D2', then with probabihty we have {D[,D2} = {-Di, D2}, and so if ^ = o(l/n^), 
we have with certainty that for all events E: 

Fj:[A{D[) £ E]< e'Pr[A(L>2) G E]. 

Since this holds for all pairs of neighboring databases, A satisfies e-differential privacy. □ 

Definition 7.4. Define the mirrored mod m function as follows: 

jp / \ f ^ mod m, if x mod 2m < m; 

™ |— x — 1 mod m, otherwise. 

'We get fe-differential privacy for Di and D2 that differ in t elements. 
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For a database D C {0, 1}", define the query 

n 

Q^{D) = F^{Y^D[i\). 

i=l 

Note that the global sensitivity of any query Qm satisfies GSq.^^ < 1. Therefore, the mechanism A that 
answers queries Qn by A{D, Qm) = Qm{D) + Z where Z is drawn from Lap(l/e) satisfies e-differential 
privacy. 

Theorem 7.5. There exist mechanisms A that satisfy e-differential privacy, but do not satisfy (e, l3)-distributional 
privacy for any e<l,/3 = o(l) (that is, for any meaningful values of€,f3). 

Proof. Consider databases with elements drawn from X = {0, 1}" and the query Q2/e- As observed 
above, a mechanism A such that A{D, Qi) = Qi{D) + Z for Z ~ Lap(l/e) satisfies e-differential privacy 
for any i. Note however that with constant probability, two databases Di,D2 drawn from S = X have 
\Q2/e{Di) — Q2/e{D2)\ > V^- Therefore, for any output x, we have that with constant probability over 
draws of two S neighbors Di and D2 : 

Fr[A{Di,Q2/,)=x] ^ \Q^^^^D,)-QyAD2)\ 

FT[A{D2,Q2/e)=^] 



Therefore the mechanism does not satisfy (1, o(l))-distributional privacy. □ 

Although there are simpler functions for which preserving distributional privacy requires more added 
noise than preserving differential privacy, the mirrored-mod function above is an example of a function 
for which it is possible to preserve differential privacy usefully, but yet impossible to reveal any useful 
information while preserving distributional privacy. 

We note that in order for distributional privacy to imply differential privacy, it is important that in the 
definition of distributional privacy, database elements are drawn from some distribution V without replace- 
ment. Otherwise, for any non-trivial distribution, there is some database that is drawn with probability 
at most 1/2", and we may modify any distributional-privacy preserving mechanism A such that for every 
query Q, A{D^:, Q) = D^, and for any Di ^ D^, A{Di, Q) behaves as before. Since this new behavior 
occurs with probability < (5 over draws from D for {3 = 0(l/2"), ^4 still preserves distributional privacy, 
but no longer preserves differential privacy (which requires that the privacy guarantee hold for every pair of 
neighboring databases). 

8 Conclusions and Open Problems 

In this paper we have shown a very general information theoretic result: that small nets are sufficient to 
certify the existence of accurate, differentially private mechanisms for a class of queries. For counting 
queries, this allows algorithms which can accurately answer queries from a class C given a database that is 
only logarithmic in the size of C, or linear its VC-dimension. We then also gave an efficient algorithm for 
releasing the class of interval queries on a discrete interval, and for releasing large-margin halfspace queries 
in the unit sphere. 

The main question left open by our work is the design of algorithms which achieve utility guarantees 
comparable to our net mechanism, but have running time only polynomial in n, the size of the input database. 
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This question is extremely interesting even for very specific classes of queries. Is there such a mechanism 
for the class of conjunctions? For the class of parity queries? 
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